Skip to content

New feature: collect Pthread IDs into DXT records #1102

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

New feature: collect Pthread IDs into DXT records #1102

wants to merge 5 commits into from

Conversation

@wkliao
Copy link
Collaborator

@wkliao wkliao commented Nov 10, 2025

This PR enables Darshan to collect pthread IDs (type of an unsigned long) into DXT records.

See #1066 for more discussion. Note that #1066 also mentions an extra annotation, in addition to pthread ID, but this PR only adds pthread IDs, without the extra annotation. Adding the extra annotations will be in a separate PR.

An example of text output from darshan-dxt-parser is shown below.
It has "Pthread-ID" added as the right-most column.

# Module    Rank  Wt/Rd  Segment          Offset          Length    Start(s)      End(s)     Pthread-ID
 X_POSIX       2  write        0              16               8      0.0022      0.0022     140498826330816

For older versions:

# Module    Rank  Wt/Rd  Segment          Offset          Length    Start(s)      End(s)   [OST]     Pthread-ID
 X_POSIX       0  write        0            2048          262144      0.0300      0.0331   [106]     -1

wkliao and others added 2 commits November 10, 2025 13:29
@wkliao
bump up DXT_POSIX_VER to 2 and DXT_MPIIO_VER to 3
36f584a
@roblatham00 @wkliao
add support for annotating DXT segment with pthread_id
ed97f54 
* Check DXT_POSIX and DXT_MPIIO versions of input log file
  The version numbers determine whether the input file contains the
  extra info field. For DXT_POSIX_VER < 2 and DXT_MPIIO_VER < 3, the
  buffer passed to darshan_log_get_mod() must not contain the space
  of pthread_id. Otherwise read contents will be incorrect.
@wkliao
Copy link
Collaborator Author

wkliao commented Nov 12, 2025

HI, @GueroudjiAmal and @infispiel

@carns suggested me to reach out to you for an advise about this PR.

As the text outputs generated from darshan-dxt-parser is organized into columns, this PR
adds a new column at the end, showing the pthread IDs. There can be two options for
presenting its values. One is to use the ID returned from a call to pthread_self() which may
be a type of unsigned long integer or a memory pointer address depending on the systems.
The other option is a number relative to the total number of threads per rank. Please let us
know which one works best for you.

Below shows two examples outputs. Note this only affects the outputs of darshan-dxt-parser.
The darshan log files (i.e. the binary files) still store the pthread IDs returned from pthread_self().

Option 1:

# DXT, rank: 1, hostname: acfs.ece.northwestern.edu
# DXT, write_count: 4, read_count: 3
# DXT, mnt_pt: /homes, fs_type: autofs
# Module    Rank  Wt/Rd  Segment          Offset          Length    Start(s)      End(s)   Pthread-ID
 X_POSIX       1  write        0              30              30      0.0017      0.0017   139906430563008
 X_POSIX       1  write        1             200              10      0.0019      0.0020   139906339825408
 X_POSIX       1  write        2             180              10      0.0020      0.0020   139906356610816
 X_POSIX       1  write        3             190              10      0.0020      0.0020   139906348218112
 X_POSIX       1   read        0              70              10      0.0019      0.0019   139906348218112
 X_POSIX       1   read        1              60              10      0.0019      0.0019   139906356610816
 X_POSIX       1   read        2              80              10      0.0019      0.0019   139906339825408

and option 2:

# DXT, rank: 1, hostname: acfs.ece.northwestern.edu
# DXT, write_count: 4, read_count: 3
# DXT, mnt_pt: /homes, fs_type: autofs
# Number of threads: 4
# Module    Rank  Wt/Rd  Segment          Offset          Length    Start(s)      End(s)   Pthread-ID
 X_POSIX       1  write        0              30              30      0.0022      0.0022      0
 X_POSIX       1  write        1             180              10      0.0023      0.0024      1
 X_POSIX       1  write        2             200              10      0.0024      0.0024      2
 X_POSIX       1  write        3             190              10      0.0024      0.0024      3
 X_POSIX       1   read        0              60              10      0.0023      0.0023      1
 X_POSIX       1   read        1              70              10      0.0024      0.0024      3
 X_POSIX       1   read        2              80              10      0.0024      0.0024      2

@GueroudjiAmal
Copy link

GueroudjiAmal commented Nov 13, 2025

Hi,
In my opinion it is better to have the pthread ID returned by pthread_self() , to further trace the thread/IO mapping and eventually correlate that with other thread related tasks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@roblatham00 roblatham00 Awaiting requested review from roblatham00

@orcunyildiz orcunyildiz Awaiting requested review from orcunyildiz

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

pydarshan

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@wkliao @GueroudjiAmal @roblatham00